Comparative evaluation of research vs. Online MT systems

This paper reports MT evaluation experiments that were conducted at the end of year 1 of the EU-funded CoSyne 1 project for three language combinations, considering translations from German, Italian and Dutch into English. We present a comparative evaluation of the MT software developed within the project against four of the leading free webbased MT systems across a range of state-of-the-art automatic evaluation metrics. The data sets from the news domain that were created and used for training purposes and also for this evaluation exercise, which are available to the research community, are also described. The evaluation results for the news domain are very encouraging: the CoSyne MT software consistently beats the rule-based MT systems, and for translations from Italian and Dutch into English in particular the scores given by some of the standard automatic evaluation metrics are not too distant from those obtained by wellestablished statistical online MT systems

Modelling the analysis of translation memory use and post-editing of raw machine translation output: A pilot study of trainee translators' perceptions of difficulty and time effectiveness

This paper describes a pilot study undertaken to propose a model for the analysis of the respective impact of translation memory (TM) use and full post-editing (PE) of raw machine translation (MT) output on the level of difficulty perceived and on the time needed by trainee translators. Six Italian MA-level translation students were asked to produce high-quality target texts when translating semi-specialised material from English into their native Italian. For this experiment, we proposed a model of data triangulation in which we measured the time taken to complete the tasks and we collected data on their translation with TM software and PE processes by means of think-aloud protocols (TAPs) and retrospective interviews. We studied the extent to which the number of translation solutions regarded as correct influenced, on the one hand, the perception of difficulty associated with the translation strategies employed and, on the other, the duration of the translation and PE tasks. Using a TM led to a reduction of the difficulty perceived and of the time employed by the participants as a result of the increased correct translation solutions provided. In contrast, a reduction was not observed when participants post-edited raw MT output. Further factors were assumed to influence the translation and PE processes of the students, especially their attitudes towards the translation technologies being use

Integrating machine translation into MOOCS

This paper presents TraMOOC (Translation for Massive Open Online Courses), a European research project developed with the intention of empowering international learners in the digital multilingual world by providing reliable machine translation (MT) specifically tailored to MOOCs from English into 11 languages (Bulgarian, Chinese, Croatian, Czech, Dutch, German, Greek, Italian, Polish, Portuguese, and Russian). The paper describes how the project is addressing the challenges involved in developing an innovative, high-quality MT service for producing accurate translations of heterogeneous multi-genre MOOC materials, encompassing subtitles of video lectures, assignments, tutorials, and social web text posted on student blogs and fora. Based on the results of a large-scale and multi-method evaluation conducted as part of the TraMOOC project, we offer a reflection on how to best integrate state-of-the-art MT into MOOC platforms. The conclusion summarizes the key lessons learned, that can be applied by the wider community of international professionals with an interest in the multilingual aspects of innovative education and new learning technologies

Evaluating MT for massive open online courses: a multifaceted comparison between PBSMT and NMT systems

This article reports a multifaceted comparison between statistical and neural machine translation (MT) systems that were developed for translation of data from Massive Open Online Courses (MOOCs). The study uses four language pairs: English to German, Greek, Portuguese, and Russian. Translation quality is evaluated using automatic metrics and human evaluation, carried out by professional translators. Results show that neural MT is preferred in side-by-side ranking, and is found to contain fewer overall errors. Results are less clear-cut for some error categories, and for temporal and technical post-editing effort. In addition, results are reported based on sentence length, showing advantages and disadvantages depending on the particular language pair and MT paradigm

Evaluating MT for massive open online courses

This article reports a multifaceted comparison between statistical and neural machine translation (MT) systems that were developed for translation of data from massive open online courses (MOOCs). The study uses four language pairs: English to German, Greek, Portuguese, and Russian. Translation quality is evaluated using automatic metrics and human evaluation, carried out by professional translators. Results show that neuralMTis preferred in side-by-side ranking, and is found to contain fewer overall errors. Results are less clear-cut for some error categories, and for temporal and technical post-editing effort. In addition, results are reported based on sentence length, showing advantages and disadvantages depending on the particular language pair and MT paradigm

Progress of the PRINCIPLE project: promoting MT for Croatian, Icelandic, Irish and Norwegian

This paper updates the progress made on the PRINCIPLE project, a 2-year action funded by the European Commission un-der the Connecting Europe Facility (CEF) programme. PRINCIPLE focuses on col-lecting high-quality language resources for Croatian, Icelandic, Irish and Norwe-gian, which have been identified as low-resource languages, especially for build-ing effective machine translation (MT) systems. We report initial achievements of the project and ongoing activities aimed at promoting the uptake of neural MT for the low-resource languages of the project

Introducing the Digital Language Equality Metric: Technological Factors

This paper introduces the concept of Digital Language Equality (DLE) developed by the EU-funded European Language Equality (ELE) project, and describes the associated DLE Metric with a focus on its technological factors (TFs), which are complemented by situational contextual factors. This work aims at objectively describing the level of technological support of all European languages and lays the foundation to implement a large-scale EU-wide programme to ensure that these languages can continue to exist and prosper in the digital age, to serve the present and future needs of their speakers. The paper situates this ongoing work with a strong European focus in the broader context of related efforts, and explains how the DLE Metric can help track the progress towards DLE for all languages of Europe, focusing in particular on the role played by the TFs. These are derived from the European Language Grid (ELG) Catalogue, that provides the empirical basis to measure the level of digital readiness of all European languages. The DLE Metric scores can be consulted through an online interactive dashboard to show the level of technological support of each European language and track the overall progress toward DLE